Repository navigation
FEAT: Scanner dataset selection: required, linked and first datasets (-5.5) - #4476
Richard Lundeen (richlundeen) wants to merge 4 commits into
Conversation
…(-5.5) Carves the generation-independent scanner and dataset changes out of the GUI generation layer (microsoft#4135), so they can be reviewed on their own: - ScenarioDatasetOptions.min_selected, validated against max_selected and enforced when a request selects datasets ("Select a dataset." / "Select at least N datasets."). foundry.assert_policy declares 1. - Scenario.get_dataset_options() reports inline seeds/seed_groups as fixed sources with no required selection. - Scanner launch page: Browse datasets moves into the Datasets header, the Datasets heading and picker are marked required, a launch without a required dataset reports it on the focused picker, a scenario with no default datasets can choose its first one, and Reset datasets shows only after source or per-dataset limit edits. - Repeatable ?dataset= links preselect datasets exactly as named; a scanner that takes no datasets blocks the launch with a link that drops them. - Estimate and launch errors append the server's 422 field messages. - Dataset catalog gains an Origins facet persisted as origin URL parameters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
An incompatible dataset link can leave a fixed-source scenario with a backend-rejected selection and no available removal, reset, or recovery action.
1 open issue
What changed in this PR
This PR improves dataset selection in PyRIT’s Scanner, allowing scenarios without defaults to select their first dataset and supporting dataset preselection through launch links.
Changes:
- Adds minimum dataset requirements and preserves fixed sources for inline-seed scenarios.
- Adds linked dataset selection, picker validation, and clearer API error messages.
- Adds dataset-origin filtering and a shared Browse datasets action.
| File | Description |
|---|---|
| tests/unit/scenario/foundry/test_assert_policy.py | Tests required dataset selection. |
| tests/unit/scenario/core/test_scenario_dataset_overrides.py | Tests selection bounds and inline-source restrictions. |
| pyrit/scenario/scenarios/foundry/assert_policy.py | Declares the minimum dataset requirement. |
| pyrit/scenario/core/scenario.py | Reports inline sources as fixed. |
| pyrit/scenario/core/dataset_overrides.py | Enforces minimum dataset counts. |
| pyrit/models/scenario_dataset.py | Adds validated minimum-selection metadata. |
| frontend/src/utils/routeParams.ts | Parses repeatable dataset link parameters. |
| frontend/src/utils/routeParams.test.ts | Tests exact dataset-name parsing. |
| frontend/src/types/index.ts | Exposes minimum-selection metadata. |
| frontend/src/services/errors.ts | Formats API field-validation messages. |
| frontend/src/services/errors.test.ts | Tests validation-message formatting. |
| frontend/src/hooks/useScenarioRunEstimate.ts | Displays detailed estimate errors. |
| frontend/src/hooks/useScenarioLaunch.ts | Displays detailed launch errors. |
| frontend/src/hooks/useScenarioLaunch.test.tsx | Tests launch validation errors. |
| frontend/src/components/Scenarios/scenarioLaunchRequest.ts | Validates required and unsupported linked datasets. |
| frontend/src/components/Scenarios/scenarioLaunchRequest.test.ts | Tests dataset request validation. |
| frontend/src/components/Scenarios/scenarioLaunchForm.ts | Seeds dataset sources from links. |
| frontend/src/components/Scenarios/ScenarioDetail.tsx | Integrates linked datasets, required selection, and recovery. |
| frontend/src/components/Scenarios/ScenarioDetail.test.tsx | Tests dataset launch workflows. |
| frontend/src/components/Scenarios/ScenarioDetail.styles.ts | Styles section-header actions. |
| frontend/src/components/Scenarios/ScenarioDatasetTable.tsx | Enables first selection and picker-level errors. |
| frontend/src/components/Scenarios/ScenarioDatasetTable.test.tsx | Tests empty, fixed, and subset selections. |
| frontend/src/components/Scenarios/DatasetActions.tsx | Adds the shared catalog navigation action. |
| frontend/src/components/Scenarios/DatasetActions.test.tsx | Tests catalog navigation. |
| frontend/src/components/Scenarios/DatasetActions.styles.ts | Styles dataset actions and touch targets. |
| frontend/src/components/Datasets/datasetQuery.ts | Adds URL-backed origin filtering. |
| frontend/src/components/Datasets/datasetQuery.test.ts | Tests origin filtering and serialization. |
| frontend/src/components/Datasets/DatasetCatalog.tsx | Adds the Origins filter control. |
| frontend/src/components/Datasets/DatasetCatalog.test.tsx | Tests origin-filter interaction. |
| frontend/e2e/numeric-controls.spec.ts | Updates reset-button expectations. |
| doc/gui/0_gui.md | Documents dataset selection, links, and origins. |
| .github/instructions/scenarios.instructions.md | Documents dataset capability declarations. |
Review effort: Balanced
| preset, | ||
| seededTechniques: techniqueSeed.techniques, | ||
| initialRun, | ||
| datasetNames: supportsDatasets && linkedDatasetNames.length > 0 ? linkedDatasetNames : undefined, |
There was a problem hiding this comment.
Fixed in commit 77b7a33. Linked ?dataset= names now go only to a scanner whose sources can be chosen: it must take dataset_config, support overrides, and not report fixed sources. The form now uses this check for linkedDatasetProblem and for the seeding at this line. A scanner with fixed sources, such as garak.api_key, does not seed the linked names. It shows "This scenario can't take the linked datasets: ..." with the "Open without the linked datasets" link, and Launch scan stays disabled. A new ScenarioDetail test covers a fixed-source scanner opened with ?dataset=.
Verified on published commit caa9d1a101eb9cef60795a47feee3dd6923f00cc.
Enforce a scenario's declared min_selected in the library for every named selection, with or without dataset overrides, and remove the frontend and AssertPolicy copies of the rule so the GUI shows the server's message. Block linked datasets on scanners with fixed sources, open Browse datasets in a new tab from one shared action, show Reset datasets only for real edits, name the field in validation messages, mark a required dataset once, share origin labels, show the dataset minimum in the CLI listing, and keep the docs and instructions accurate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A scenario that declares min_selected and has no default datasets has no default run to size. Return an unavailable default estimate with the selection message instead of raising, so the catalog shows the reason without logging a failure on every load. Configured estimates and runs still reject the selection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aunch - Size the default run against the default configuration's dataset options, so a reused scenario instance returns the unavailable estimate instead of raising. - Label a 400/422 estimate as a configuration to complete, not a backend failure. - Focus the launch error banner after a rejected launch so its reason scrolls into view. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Part -5.5 of the ASSERT stack. Design and how to use it: ASSERT in PyRIT.
This layer holds the scanner and dataset changes from #4135 that don't need dataset generation, so they can be reviewed on their own. It targets
maindirectly; #4135 will be rebuilt on top of it.Description
A scenario can now say it needs at least one dataset, and the scanner launch page lets you choose the first dataset, link datasets in by URL, and see why a launch was rejected. On
main,foundry.assert_policyhas no default datasets and the launch page offers no way to pick one, so it can't be launched from the GUI.ScenarioDatasetOptionsgainsmin_selected, the counterpart ofmax_selected. It must be positive and no larger thanmax_selected, andvalidate_dataset_selectionenforces it with "Select a dataset." / "Select at least N datasets.".foundry.assert_policydeclaresmin_selected=1. The value reaches the GUI through the existing catalog and estimate models.Scenario.get_dataset_options()reports inlineseeds/seed_groupsasfixedsources, because named sources can't replace them, and drops any declared minimum. Among registered scenarios this affects onlytechnique.execute, which takes no dataset input.min_selectedmarks the section and picker required; launching without a dataset shows "Select a dataset." on the picker and focuses it.?dataset=parameters preselect datasets exactly as named, replacing the default or preset datasets and keeping their limits and filters. The server validates the names. On a scanner that takes no datasets, linked names block the launch with a link to open the page without them.originparameters.foundry.assert_policybefore, onmain: no dataset picker, so it can't be launched.After: the Datasets section is required, with Browse datasets in the header and a picker for the first dataset.
After, Launch scan without a dataset: "Select a dataset." on the focused picker.
After, opened with
?dataset=jailbreak_templates: the dataset is preselected.More screenshots
Ordinary scanner (
foundry.red_team_agent), before onmain: Browse datasets is a link under the table, and Reset datasets shows on untouched defaults.After, untouched: Browse datasets in the header, no Reset datasets.
After adding a dataset: Reset datasets appears.
Blank linked name (
foundry.red_team_agent?dataset=%20): the run estimate shows the server's message.Scanner that takes no datasets (
technique.execute?dataset=jailbreak_templates): the launch is blocked with a link that drops the linked datasets.Dataset catalog: the Origins filter, then filtered to Local files.
foundry.assert_policyDatasets section at 390 px:Screenshots use the repo's offline end-to-end backend with no provider or model calls. Launch-page controls outside the Datasets section aren't from this PR.